System and Method for Determining Mobile Telephone Voice Quality in a Live Network

ABSTRACT

A system and method for determining the quality of communications provided by a plurality of mobile telephones is provided. The method may comprise, for example, receiving one or more telephone voice signals from each mobile telephone; storing the one or more telephone voice signals in a memory; concurrently with aid receiving telephone voice signals from a mobile telephone, receiving one or more reference device voice signals from a reference device substantially co-located with the mobile telephone; storing the one or more reference device voice signals in a memory; determining voice quality data for the telephone voice signals; determining voice quality data for the reference device voice signals concurrently received with the telephone voice signals; and normalizing the voice quality data of the telephone voice signals from each mobile telephone based on the voice quality data of the reference device voice signals concurrently received with the telephone voice signals.

CROSS REFERENCE TO RELATED APPLICATIONS

This application is a continuation of, and claims priority to, U.S.patent application Ser. No. 12/037,310, filed Feb. 26, 2008, which ishereby incorporated by reference in its entirety for all purposes.

FIELD OF THE INVENTION

The present invention generally relates to systems and methods forevaluating communication devices, and more particularly to systems andmethods for evaluating and comparing voice quality of mobile telephonesin live networks.

BACKGROUND OF THE INVENTION

Mobile telephone devices have become ubiquitous in our society. Unlikeconventional landline telephony services, which typically operate in ahome or office and are delivered via a wire and enjoy a highlyconsistent transmission medium, mobile telephone devices are subjectedto use under varying radio environments which result in a highly varyingspeech quality to the end user. One challenge to those designing mobiletelephone devices is to design the telephone devices to provide thedesired speech and data quality even when the user is using thetelephone in challenging and changing radio environments. Further,wireless network operators also want the users of their network to usetelephone devices that provide adequate communication quality in allradio environments to ensure that the user has a satisfactory experienceusing their wireless network. Thus, there are numerous parties whodesire to test the quality of communications provided by mobiletelephone devices.

There are, however, a wide variety of telephone devices used forcommunicating over wireless mobile telephone networks. With theproliferation of mobile telephone devices, many designs of telephoneshave evolved. The different designs of telephone devices result indifferent performance characteristics for each telephone device. Variousdesign characteristics may impact the quality of the speech and dataservices provided by a telephone device and its ability to providecommunications in varying radio environments. For example, the radiofront-end of a telephone device, which drives, in part, the radiatedperformance (a devices ability to receive and transmit radio signals) ofthe device may positively or negatively impact the quality ofcommunications in various radio environments. Another factor may be thedevice's capability to cancel interfering radio signals from wantedradio signals in order to increase the signal-to-noise ratio and therebyimprove the quality of communications. Other design factors include (a)the performance of the device's digital signal processor, (b) the designof the device's operating system and associated applications includingthe handling of TCP/IP communication. As a result, the many differentmobile telephone devices have varying performance characteristics due totheir design. Thus, different telephone devices operating in the sameradio environment provide different qualities of communication.

In addition, another challenge to mobile telephone device manufacturers,and to wireless carriers, is how to comparatively evaluate differentmodel phones across different locations, at different times, and underthe differing radio conditions present in live mobile networks. Thepresent invention provides methods and systems to objectively determinethe voice quality of different mobile telephones in varying environmentseven when tested at different times. These and other advantages may beprovided by one or more embodiments of the present invention.

SUMMARY OF THE INVENTION

The present invention provides a system, product, and method fordetermining the quality of communications provided by a plurality ofmobile telephones. In one embodiment, the method may include receivingone or more telephone voice signals from each mobile telephone over atime period at one or both of a voice server or field voice quality testequipment. Concurrently with receiving telephone voice signals from eachmobile telephone, receiving one or more reference device voice signalsfrom a reference device substantially co-located with the mobiletelephone. The method further includes determining the voice quality ofthe telephone voice signals and the reference device voice signalsconcurrently received with the telephone voice signals; determining oneor more telephone statistical parameters based on the voice quality ofthe telephone voice signals; determining one or more reference devicestatistical parameters based on the voice quality of the referencedevice voice signals concurrently received with the telephone voicesignals; and normalizing the one or more telephone statisticalparameters of the telephone voice signals from each mobile telephonebased on the one or more reference device statistical parameters of thereference device voice signals that were concurrently received with thetelephone voice signals. The statistical parameters may include (1) anaverage of the voice quality samples; (2) a standard deviation of thevoice quality samples; (3) a mean of the voice quality samples; (4) apercentage of the voice quality samples above a threshold; and (5) apercentage of the voice quality samples below a threshold. The methodmay also include collecting radio parameter data during the test andcomparing normalized statistical parameters of telephones tested insimilar radio conditions.

The invention will be better understood by reference to the followingdetailed description taken in conjunction with the accompanyingdrawings.

BRIEF DESCRIPTION OF THE DRAWINGS

The invention is further described in the detailed description thatfollows, by reference to the noted drawings by way of non-limitingillustrative embodiments of the invention, in which like referencenumerals represent similar parts throughout the drawings. As should beunderstood, however, the invention is not limited to the precisearrangements and instrumentalities shown. In the drawings:

FIG. 1 is a block diagram of an example system for determining the voicequality performance provided by one or more mobile telephone devicesaccording to an example embodiment of the present invention;

FIG. 2 illustrates an example method for determining voice qualityperformance according to an example embodiment of the present invention;and

FIG. 3 illustrates another example method for determining voice qualityperformance according to an example embodiment of the present invention.

DETAILED DESCRIPTION OF ILLUSTRATIVE EMBODIMENTS

In the following description, for purposes of explanation and notlimitation, specific details are set forth, such as particular networks,communication systems, computers, terminals, devices, components,techniques, telephone devices, mobile telephones, accessory devices,simulators, ear pieces, headsets, telephone handsets, data and networkprotocols, software products and systems, operating systems, developmentinterfaces, hardware, etc. in order to provide a thorough understandingof the present invention.

However, it will be apparent to one skilled in the art that the presentinvention may be practiced in other embodiments that depart from thesespecific details. Detailed descriptions of well-known networks,networks, communication systems, computers, telephone devices, mobiletelephones, accessory devices, simulators, ear pieces, headsets,telephone handsets, terminals, devices, components, techniques, data andnetwork protocols, software products and systems, developmentinterfaces, operating systems, and hardware are omitted so as not toobscure the description of the present invention.

According to an embodiment of the present invention, the voice qualityperformance of a mobile telephone device, such as a mobile telephone, isevaluated in a live network. As used herein, “mobile telephone” means atelephone configured to provide voice communication services over amobile telephone network. Other telephone devices include mobiletelephone accessories (e.g., a wired or wireless) such as an earpiece,headset, speaker phone (e.g., that includes a microphone and which maybe, for example, in an automobile, or other device), or other suchdevice. A mobile telephone, also sometimes commonly referred to as acell telephone, is a long-range, mobile electronic device used formobile communications. In addition to providing the standard voicefunction of a telephone, many mobile telephones may support additionalservices such as SMS for text messaging, email, FTP, UDP, packetswitching for access to the Internet, and MMS (Multimedia MessagingService) for sending and receiving photos and video. A conventionalmobile telephone may wirelessly communicate via a cellular network ofbase stations (cell sites), which is connected to the public switchedtelephone network (PSTN). A mobile telephone device thus may include amobile telephone alone, or in combination with an accessory.

The present invention may be used to evaluate voice communications ofmobile telephone devices for various mobile communication standards andnetwork protocols. Voice quality may be evaluated for both transmittedand received voice communications. The mobile telephone devices undertest, as well as a reference mobile telephone device (referred to as a“reference device”), are coupled to voice quality test equipment (VQTE)in the field. Calls are concurrently placed from the mobile telephonedevices under test and the reference device, to a voice server. The VQTEdetermines the voice quality of voice communications received by thetelephones under test and the reference device. The voice serverdetermines the voice quality of communications received from therespective mobile telephone devices under test and the reference device.Bi-direction test voice communications are communicated while thedevices under test and the reference device move throughout ageographical area of the live network. Thus, the voice quality may beevaluated for the devices under test and reference device at variousgeographic locations within one or more networks. Tests may be performedfor a plurality of different telephones at different times and differentlocations. Various statistical parameters of a plurality of voicesamples may be calculated and stored for the reference device and theone or more devices under test, including the average voice qualityscore, the standard deviation of the voice quality score, the percentageof voice quality scores above or below a selected performance thresholdscore. The statistical parameters of the voice samples of each deviceunder test may be normalized based on the statistical parameters of thereference device that concurrently communicates voice signals to therebynormalize the test data of different telephones at different locationsand under different environmental and network conditions. As a result,the performance of multiple mobile telephone devices may be meaningfullycompared.

In some embodiments, a device (e.g., another mobile telephone) also maybe used to determine (e.g., measure or estimate) one or more radioparameters experienced by the mobile telephone devices and referencedevice during the test. Such device may be coupled to radio parametercollection equipment. In some embodiments the voice quality testequipment and radio parameter collection equipment may be a computerwhich includes software for evaluating voice quality and receiving radioparameter data. Exemplary radio data parameters may include one or moreof the following: signal to noise ratio, bit error rate, frame errorrate, channel codec, received-signal level, downlink channel, channeltype, handover statistics, and speech energy.

In an example method embodiment of the present invention, voice qualitysamples and radio parameter data may be collected while driving avehicle in a geographical area of interest. Voice quality samples may beobtained for the device(s) under test and the reference devicethroughout the test at various times. A time stamp and location data maybe stored in memory for each voice quality sample and periodically foreach portion of radio parameter data. The time stamp may be used toselect or derive radio parameter data for specific voice qualitysamples. In some embodiments the radio parameter data device may be fromthe reference device or a device under test. In other embodiments, aseparate measuring device may be used.

The voice server and/or the VQTE may organize or associate the one ormore normalized telephone statistical parameters based on the radioparameter data for the time period during which the telephone signalswere received. The system may then provide a comparison of thenormalized telephone statistical parameters that includes the normalizedtelephone statistical parameters grouped according to the radioconditions in which the telephone voice signals were received.

Test System Environment

FIG. 1 shows an example embodiment of a test system environment 100 fortesting one or more mobile telephone devices 102. The test systemenvironment 100 may include a reference mobile telephone device 104 (thereference device), and one or more mobile telephone devices 102 a,b tobe tested (the devices under test). The reference device 104 and thedevices under test 102 are electrically connected (via their headsetjack) to voice quality test equipment 106. The test system environment100 also may include a voice server 108 and communication network 110.The reference device 104 and devices under test 102 communicate with thevoice server 108 through the communication network 110. In someembodiments the test system environment 100 also may include one or moredevices 114 for collecting radio parameter data.

The reference device 104 may be any suitable mobile telephone device andpreferably comprises the same model of telephone during all tests (butnot necessarily the same physical device). The reference device 104performs the same test procedures as the devices under test 102 andprovides, in effect, a baseline for comparing performance of variousdevices 102 that are tested. In some implementations, different groupsof mobile telephones may be tested with, and compared to, differentreference devices 104. Note that although different test groups may usedifferent reference devices 104, when any mobile telephone device iscommon to multiple groups, the results may be meaningfully compared forthose groups (e.g., by rescaling the results using the common device asthe reference device).

The VTQE 106 may comprise a laptop computer, handheld computer or otherportable computing device capable of executing software to implementprocesses of the test scenarios. One function may include determiningand storing voice quality samples, as well as determining statisticalparameters (e.g., average, mean etc.) of the samples, for the referencedevice 104 and devices under test 102. Another function may includestoring radio parameter data obtained by the collection device 114during testing.

The voice server 108 may be a computing device executing software forperforming voice server functions. For example, the voice server 108 mayperform test procedures, including transmitting and receiving voicecommunications and scoring voice quality of received voicecommunications. The voice server 108 also may store voice qualitysamples, as well as determining statistical parameters (e.g., average,mean etc.) of the samples, for the reference device 104 and devicesunder test 102. obtain.

The communication network 110 includes at least a mobile telephonecommunication network 116. The mobile telephone network 116 may be aradio network made up of a number of radio cells (or just cells) eachserved by a fixed transmitter, known as a cell site or base station.These cells are used to cover different areas in order to provide radiocoverage over a wider area than the area of one cell. An exemplarymobile telephone network 116 may be based on any of varioustelecommunication standards, such as AMPS, D-AMPS, CDMA2000, GSM, GPRS,EV-DO, UMTS, EDGE, HSCSD, HSPA, FOMA, CDMA, WiMAX, G1, G1.5, G2, and G3.

In this example, the voice server 108 is coupled to the mobile telephonenetwork 116 via the public switched telephone network (PSTN) 118. Thereference device 104 and devices under test 102 each transmit andreceive communications through the mobile telephone network 116. Inother embodiments the voice server 108 may be coupled directly to themobile telephone network 116. In other embodiments the communicationsnetwork 110 also may include additional wired or wireless networks, suchas an internet protocol (IP) network, a broadband communication network,a VoIP network, and/or another wired or wireless network. Thus, thereference device 104 and devices under test 102 may communicate with thevoice server 108 along a communication path which includes the mobiletelephone network 116 alone or in combination with one or more othernetworks.

Voice Quality Test Methods

FIG. 2 illustrates an example process 200 for evaluating voice qualityperformance in a live network according to an embodiment of the presentinvention. During testing, the devices under test 102 and referencedevice 104 may be located in a vehicle (moving or stationary) within theoperating area of one or more mobile telephone networks 116. Forexample, the devices 102, 104 may be situated within a vehicle thattravels along a substantially random or a predetermined route. Thedevices 102, 104 may be electrically coupled to the VQTE 106 whichstores the voice source files to be supplied to each device 102, 104 fortransmission and stores and processes the received voice communicationsto determine a voice quality (e.g., MOS). At step 202, each of thedevices 102, 104 establishes a communication link with the voice server108 through the mobile telephone network 116 by placing a call to thevoice server 108. The voice server 108 may have multiple landlinetelephone connections to receive multiple calls. The telephone calls maybe placed to the voice server 108 by a test operator, or automaticallyin response to a command from the VQTE 106.

At step 204, voice communications may be sent from the reference device104 and devices under test 102 to the voice server 108, and from thevoice server 108 to the reference device 104 and device(s) under test102. For example, the voice source files stored on the VQTE 106 may bereproduced (perhaps concurrently) as voice inputs to the earpiece jackof the reference device 104 and devices under test 102 and aretransmitted as voice data through the mobile telephone network 116 tothe voice server 108. Similarly, the voice server 108 also may includestored source voice files that are reproduced as voice input to therespective landlines and communicated as voice data through thelandlines and mobile telephone network 116 to be received (perhapsconcurrently) by the reference device 104 and devices under test 102.

Voice communications may be sent and received in a half-duplex mode. Forexample, voice communications may be sent and received in an alternatingmanner every five to thirty seconds and may generally model a telephoneconversation. The duration period for the voice communication may vary.Accordingly, voice communications may be received at each of thereference device 104 and devices under test 102 intermittently during agiven test. Similarly, voice communications may be received at the voiceserver 108 from each of the reference device 104 and devices under test102 intermittently during the test.

The reference device 104 and devices under test 102 may communicatevoice communications concurrently. As a result, multiple communicationlinks may be established concurrently between the voice server 108 andthe respective devices 102, 104. For a given device 102, 104 the testingmay occur over the course of one telephone call. In an alternativeembodiment the testing may occur over the course of multiple telephonecalls between the device 102, 104 and the voice server 108.

Although the voice communications are described as being generallyconcurrent, one of skill in the art will appreciate that the referencedevice 104 and devices under test 102 need not be sending and receivingsimultaneously (although they very well could) and need not establishcommunications links with the voice server that are preciselyco-extensive. Different communication timing may be used. In addition,in this example the same voice source file(s) may be reproduced to eachdevice 102 104 so that an objective comparison can be made when thevoice quality is determined. In other embodiments, different voice filesmay be used. An advantage of using a common voice communication for alloutgoing communications from the field is that any variations in themeasured voice quality result at the receiving end are not likely to beattributable to a difference in the source communications.

As discussed, the reference device 104 and devices under test 102 arecoupled to the VQTE 106. During the test, the VQTE 106 and voice server108 store the incoming voice communications as voice communicationfiles. At step 206, the VQTE 106 executes an algorithm on each storedvoice communication file to determine the voice quality (e.g., todetermine a mean opinion score (MOS)) of the incoming voicecommunications for each device 102, 104. Similarly, at step 216 thevoice server 108 executes an algorithm on each voice communication fileto determine the voice quality (e.g., to determine the MOS) of theincoming voice communications from each device 102, 104. Any suitablescoring method may be employed.

As discussed, voice communications may be transmitted between the voiceserver 108 and the devices 102, 104 intermittently and each voicecommunication may be stored as a separate file. The VQTE 106 maydetermine a voice quality evaluation for each incoming voicecommunication (whether stored as a separate file or not). The resultingvoice quality assessment (e.g., MOS) for each stored voice communicationis referred to as a voice quality sample. Thus, a test of a singledevice 102, 104 may result in a plurality of voice quality samples.Alternatively, the VTQE 106 may determine a voice quality result foreach portion of a communication having a select sample length, (e.g.,every 2 second clip of a 10 second voice communication received at aspecific device 102, 104). Thus, at step 208 the VTQE 106 may store thevoice quality sample, device location, and the time and date of thevoice quality sample, along with an indication of the device 102, 104which received the voice communication.

Similarly, at step 216 the voice server 108 may determine a voicequality evaluation for each incoming voice communication (whether storedas a separate file or not) to provide a plurality of voice qualitysamples for each device 102, 104. At step 218, the voice server 108 maystore the voice quality sample, device location, and the time and dateof the voice quality sample, along with an indication of the device 102,104 which transmitted the voice communication.

At step 210 (and step 220), the voice quality samples of each device102, 104 may be processed to determine one or more statisticalparameters. For example, for each device 102, 104, the VQTE 106 and/orvoice server 108 may determine (1) an average voice quality sample; (2)a standard deviation of the voice quality samples; (3) a mean of thevoice quality samples; (4) a percentage of voice quality samples above athreshold; and (5) a percentage of voice quality samples below athreshold. The results of this processing is stored in memory of theVQTE 106 and/or voice server 108.

At step 212 (and step 222), the statistical parameters of voice qualitysamples for each device under test 102 may be normalized base on (orscaled or otherwise referenced to) the statistical parameters of voicequality samples for the reference device 104. In an example embodiment,the normalized average voice quality sample for a device under test 102may be determined by computing the average voice quality sample for thedevice 102 minus the average voice quality sample for the referencedevice 104. The normalized standard deviation may be determined bycomputing the standard deviation of voice quality samples for the deviceunder test 102 minus the standard deviation of voice quality samples forthe reference device 104. The normalized percentage of voice qualitysamples above (or below) a given threshold may be determined bycomputing the normalized percentage of voice quality samples of a device102 above (or below) the given threshold minus the normalized percentageof voice quality samples of the reference device 104 above (or below)the given threshold. The normalized voice quality mean may be determinedby computing the mean voice quality sample for the device under test 102minus the mean voice quality sample for the reference device 104. Inother embodiments and for other data and/or statistical parameters,other normalization techniques may be used such as by determining theaverage voice quality sample for a device under test minus the averagevoice quality sample for the reference device 104 divided by thestandard deviation of the average voice quality of the reference device104 as measured over a multitude of similar voice quality tests. In yetanother embodiment, other normalization techniques may include dividingthe value (or statistical parameter) for the device under test 102 bythe value (or statistical parameter) for the reference device. Anysuitable normalization technique may be used. In addition, while thisexample normalizes statistical parameters, other embodiments maynormalize other parameters such as throughput (e.g., when testing a datacommunication device).

Finally, in FIG. 1 the system is depicted as testing two devices 102 ata time and concurrently communicating voice communications with a singlereference device 104. In such an implementation, if one hundred devices102 were tested (two devices tested per test), there would be onehundred sets of voice quality samples (for the one hundred devices 102tested) and fifty sets of voice quality samples for reference device 104(one set for each test) that provides the baseline during each test. Aswill be evident to those skilled in the art, the invention may be usedto concurrently test one, two, three, or more devices 102. In thisembodiment, the statistical parameters of the voice quality samples fora telephone 102 are normalized based on the statistical parameters ofthe voice quality samples of the voice communications communicatedconcurrently by the reference device 104.

At steps 214 and 224, the normalized results may be compared fordifferent mobile telephone devices 102, which may have been tested atdifferent times and under different network and environmentalconditions. Even though the test scenarios are not the same for each ofthe tested devices 102, the results are normalized to the resultsprovided by the reference device 104 to thereby reduce the impact ofdiffering network environments and conditions.

While FIG. 2 illustrates the portions of the process being performed byboth the VQTE 106 and the voice server 108, data (e.g., stored voicefiles, the voice quality samples, and/or the statistical parameters) maybe transmitted from the VQTE 106 to the server 108 (or from server 108to the VQTE 106) and the server 108 (or the VQTE 106) may perform theprocesses yet remaining.

In some embodiments, it may be desirable to compare the normalized voicequality of different mobile telephone devices in various radioenvironments. For example, based upon weather, network load or otherfactors, the signal to noise ratio, bit error rate, and other radioparameters may vary. Some devices may perform very well in some radioenvironments, but perform poorly in others. Accordingly, it may bedesirable to compare the quality of communications provided by differentmobile telephone devices when operating in similar radio environments tothereby gain a more accurate and detailed comparison of different mobiletelephones.

Consequently, another embodiment may include additional processes tocollect radio parameter data of the network. As shown in FIG. 1, thetest environment 100 may include a device 114 for collecting radioparameter data. In one example embodiment the device 114 comprises amobile telephone that is electrically connected to the VQTE 106 andconfigured to collect data of one or more radio parameters of thenetwork 116. Example radio parameter data may include, but is notlimited to, signal to noise ratio, bit error rate, frame error rate,channel codec, received-signal level, downlink channel, channel type,handover statistics, and speech energy. The objective is to obtain radioparameter data which may be used to determine the radio environmentduring testing of a device 102 so that mobile telephone devices 102tested in similar radio environments can be compared and so that amobile telephone device 102 can be tested across a spectrum of differentradio environments.

FIG. 3 shows a flow chart of an example process that collects radioparameter data that is stored on the VQTE 106. This example process alsomay include the processes of FIG. 2. As step 302, the VQTE 106 (andvoice server 108) receive and store voice communications received by(transmitted from) the device(s) under test 102 and the reference device104. In addition, radio parameter data from the radio parameter datacollection device 114 is received and stored by the VQTE 106.

At step 304, the VQTE 106 and voice server 108 determine the voicequality for each voice communication to provide a plurality of voicequality samples, which may be performed as described with regard tosteps 208 and 218. Next, at 306 the VQTE 106 and voice server 108determine one or more statistical parameters (e.g., average voicequality sample, standard deviation, etc.) for the voice quality samplesfor the device(s) under test 102 and the reference device 104, which maybe performed as described with regard to processes 210 and 220. At step308, the VQTE 106 and voice server 108 each normalizes the one or morestatistical parameters for each device under test 102 based on the oneor more statistical parameters of the voice quality samples of thereference device 104. This may be performed as described with regard toprocesses 212 and 222.

Next, the normalized statistical parameters for each device under test102 may be further normalized, sorted, organized, categorized, and/orsummarized based on the radio environment experienced during the test atstep 308. For example, by comparing the time and date of the voicequality sample with the time and date of the radio parameter dataobtained, the radio parameter data for a particular voice quality samplemay be determined. In other words, when the time and date are thesubstantially the same, then the radio parameter data may be used tocategorize the corresponding voice quality samples. In practice, theradio parameter data may be stored in memory (e.g., in a database) inassociate with the voice quality samples. Subsequently, the operator mayperform various sorting, summarizing, normalizing, and/or categorizingof the voice quality samples of various devices based on the radioparameter data to thereby compare the statistical parameters for thevoice quality sample of devices tested under similar radio conditions,(which may result in excluding from some comparisons the statisticalparameters of voice quality samples of devices tested under dissimilarradio conditions).

It is worth noting that the radio parameter device 114 may be co-locatedwith the devices under test 102 and will likely experience radioconditions similar to those experienced by the devices under test 102.However, it is possible that a particular channel (or other aspect) ofcommunications experienced by the collection device 114 is differentthan that experienced by a device under test 102. Thus, for at leastsome radio parameters, the radio parameter data used to categorize thenormalized statistical parameters may be an estimate of what wasexperienced by the device under test 102.

At step 310, the normalized statistical parameters for each device 102are sorted based on the radio parameter data collected during the testof that device 102. For example, the normalized average voice qualityfor all the telephones 102 tested may be categorized according to theradio parameter data. The normalized statistical parameters oftelephones 102 may then be compared to provide a comparison of thenormalized statistical parameters of telephones 102 tested in similarradio conditions. For example, the normalized statistical parameters(e.g., average, standard deviation, mean, percentage above a threshold,percentage below a threshold, etc.) of telephones tested in radioconditions where the signal to noise ratio (SNR) is between 9 dB and 12dB may be compared for a plurality of mobile telephones 102. Similarly,the normalized statistical parameters of telephones 102 tested in othersimilar radio conditions may be compared such as one or more of (1)where the SNR is within a range (e.g., between 12 dB and 15 dB), (2)where the noise floor is above a predetermined threshold, (3) where thebit error rate is above a threshold, below a threshold, or within arange, and (4) within a window (or above or below of a threshold) of oneor more other radio parameters.

As discussed, the date, time, location data, and radio parameter datamay be stored in association with the voice communications received viaeach mobile telephone (stored so that it is retrievable and sortable) onthe voice server 108 or VQTE 106 to allow the operator to perform anydesired comparisons based on the collected data. In some embodiments, aset of specific metrics may be used to compare devices. A rating systemmay be defined from these metrics to give each device under test asingle score or set of scores and to facilitate one or more rankings ofthe tested telephones. While the above example has been described in thecontext of assessing the voice quality performance a mobile telephones,the invention is equally suitable for assessing the performance of datacommunications by wireless devices over a wireless data network such asmobile telephone network.

The normalized statistical parameters for all the devices tested maystored in memory (e.g., in a database) and sorted based on thecorresponding radio parameters. After a number of devices have beentested (e.g., fifty), a threshold may be determined—that comprises anacceptable score—for each statistical category or for an aggregatestatistical score. For example, for all additional devices tested, suchdevices must provide normalized statistical parameters for their voicequality samples that are above the bottom twentieth percentile in orderto have an acceptable score (e.g., receive a “passing grade”) and bepermitted on the mobile network, be accepted by the manufacture, and/orotherwise be acceptable.

The invention may comprise a computer program product stored in anon-transitory tangible computer readable medium and executable by acomputer to determine the quality of voice communications provided by aplurality of mobile telephones, comprising a first code segmentexecutable to store in a memory one or more telephone voice signalsreceived from each mobile telephone; a second code segment executable tostore in a memory one or more reference device voice signals receivedconcurrently with the telephone voice signals from a reference devicesubstantially co-located with the mobile telephone; a third code segmentexecutable to determine voice quality data of the telephone voicesignals and of the reference device voice signals; and a fourth codesegment executable to normalize the voice quality data of the telephonevoice signals received from each mobile telephone based on the voicequality data of the reference device voice signals concurrently receivedwith the telephone voice signals. The product may further comprise afifth code segment executable to store in a memory radio parameter datafor the mobile telephone determined for radio conditions during thereception of the one or more telephone voice signals; and a sixth codesegment executable to sort the normalized voice quality data of theplurality of mobile telephones based on the radio parameter data.

It is to be understood that the foregoing illustrative embodiments havebeen provided merely for the purpose of explanation and are in no way tobe construed as limiting of the invention. Words used herein are wordsof description and illustration, rather than words of limitation. Inaddition, the advantages and objectives described herein may not berealized by each and every embodiment practicing the present invention.Further, although the invention has been described herein with referenceto particular structure, steps and/or embodiments, the invention is notintended to be limited to the particulars disclosed herein. Rather, theinvention extends to all functionally equivalent structures, methods anduses, such as are within the scope of the appended claims. Those skilledin the art, having the benefit of the teachings of this specification,may affect numerous modifications thereto and changes may be madewithout departing from the scope and spirit of the invention.

1. A method of determining the quality of voice communications providedby a plurality of mobile telephones, comprising: storing in a memory oneor more telephone voice signals communicated by each of the plurality ofmobile telephones; storing in a memory one or more reference devicevoice signals communicated concurrently with the telephone voice signalsby a device substantially co-located with each mobile telephone thatcommunicated the one or more telephone voice signals; determining avoice quality data of the one or more telephone voice signalscommunicated by each mobile telephone; determining a voice quality dataof the one or more reference device voice signals communicatedconcurrently with the one or more telephone voice signals communicatedby each mobile telephone; and normalizing the voice quality data of theone or more telephone voice signals communicated by each mobiletelephone based on the voice quality data of the one or more referencedevice voice signals communicated concurrently with the one or moretelephone voice signals.
 2. The method according to claim 1, furthercomprising: for each mobile telephone, storing in a memory radioparameter data determined for radio conditions during the communicationof the one or more telephone voice signals.
 3. The method according toclaim 2, further comprising sorting the normalized voice quality data ofthe plurality of mobile telephones based on the radio parameter data. 4.The method according to claim 1, wherein: a voice quality data of theone or more telephone voice signals comprises a telephone average voicequality; a voice quality data of the one or more reference device voicesignals comprises a reference device average voice quality; and themethod further comprising: determining the difference of (1) the averagevoice quality data of the telephone voice signals of each mobiletelephone and (2) the average voice quality data of the reference devicevoice signals received concurrently.
 5. The method according to claim 1,further comprising determining whether the normalized voice quality dataof one or more telephone voice signals communicated by a mobiletelephone are above a predetermined threshold.
 6. The method accordingto claim 1, wherein the voice quality data of the one or more telephonevoice signals comprises an average voice quality sample.
 7. The methodaccording to claim 1, further comprising: determining a voice qualityvalue for each mobile telephone based on the normalized voice qualitydata of the one or more telephone voices signals communicated by eachmobile telephone.
 8. A computer program product stored in anon-transitory tangible computer readable medium and executable by acomputer to determine the quality of voice communications provided by aplurality of mobile telephones, comprising: a first code segmentexecutable to store in a memory one or more telephone voice signalscommunicated by each of the plurality of mobile telephones; a secondcode segment executable to store in a memory one or more referencedevice voice signals communicated concurrently with the telephone voicesignals by a device substantially co-located with each mobile telephonethat communicated the telephone voice signals; a third first codesegment executable to determine a voice quality data of the one or moretelephone voice signals communicated by each mobile telephone; a fourthcode segment executable to determine a voice quality data of the one ormore reference device voice signals communicated concurrently with theone or more telephone voice signals communicated by each mobiletelephone; and a fifth code segment executable to normalize the voicequality data of the one or more telephone voice signals communicated byfrom each mobile telephone based on the voice quality data of the one ormore reference device voice signals communicated concurrently with theone or more telephone voice signals.
 9. The computer program productaccording to claim 8, further comprising a sixth code segment executableto store in a memory, for each mobile telephone, radio parameter datadetermined for radio conditions during the communication of the one ormore telephone voice signals.
 10. The computer program product accordingto claim 9, further comprising a sixth code segment executable to sortthe normalized voice quality data of the plurality of mobile telephonesbased on the radio parameter data.
 11. The computer program productaccording to claim 8, wherein: a voice quality data of the telephonevoice signals comprises a telephone average voice quality; a voicequality data of the reference device voice signals comprises a referencedevice average voice quality; and the computer program product furthercomprising: a sixth code segment executable to determine the differenceof (1) the average voice quality data of the telephone voice signals ofeach mobile telephone and (2) the average voice quality data of thereference device voice signals received concurrently.
 12. The computerprogram product according to claim 8, further comprising a sixth codesegment executable to determine whether the normalized voice qualitydata of the one or more telephone voice signals of a mobile telephoneare above a predetermined threshold.
 13. The computer program productaccording to claim 8, wherein the voice quality data of the one or moretelephone voice signals comprises an average voice quality.
 14. Thecomputer program product according to claim 8, further comprising asixth code segment executable to determine a voice quality value foreach mobile telephone based on the normalized voice quality data of theone or more telephone voices signals communicated by each mobiletelephone.
 15. A method of determining the quality of communicationsprovided by a mobile telephone in a mobile telephone network,comprising: (a) receiving one or more telephone voice signalscommunicated by a mobile telephone; (b) concurrently with said receivingone or more telephone voice signals, receiving one or more referencedevice voice signals communicated by a reference device substantiallyco-located with the mobile telephone; (c) determining voice qualitysamples for the telephone voice signals; (d) determining voice qualitysamples for the reference device voice signals; (e) determining a firstaverage comprising an average of the voice quality samples of the mobiletelephone; (f)) determining a second average comprising an average ofthe voice quality samples of the reference device; (g) comparing thefirst average with the second average to provide a first normalizedaverage; and performing processes a-g for a plurality of mobiletelephones.
 16. The method according to claim 15, further comprisingcomparing the first normalized average with a normalized average of aplurality of the plurality of mobile telephones.
 17. The methodaccording to claim 15, wherein said receiving one or more telephonevoice signals communicated by a mobile telephone comprises receivingvoice signals that were transmitted through a mobile telephone networkby the mobile telephone.
 18. The method according to claim 15, furthercomprising determining associated radio parameter data for radioconditions for each of the plurality of mobile telephones during thecommunication of the one or more telephone voice signals.
 19. The methodaccording to claim 18, further comprising providing a comparison of thenormalized first average of the plurality of mobile telephones based onthe radio parameter data associated with each mobile telephone.
 20. Themethod according to claim 19, wherein the comparison comprises one ormore groupings of mobile telephones having similar associated radioparameter data.